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ABSTRACT 


A  study  was  conducted  to  determine  whether  the  Maintenance 
Training  Improvement  Program  (MTIP)  has  any  relationship  to 
unit  performance  of  the  west  coast  F-14  and  E-2  squadrons. 
Using  correlation  analysis,  the  MTIP  completion  rate  was 
compared  with  operational  measures  such  as  the  Full  Mission 
CapcdJle  (FMC)  rate,  the  niomber  of  no-defect  (malfunction  code 
A- 799)  maintenance  actions,  and  the  Direct  Maintenance 
Manhours  per  Flight  Hour  (DMMH/FH) .  A  moderate  positive 
correlation  was  found  between  the  MTIP  completion  rate  and  the 
FMC  rate,  and  a  moderately  weak  negative  correlation  was  found 
between  the  MTIP  completion  rate  and  the  number  of  A- 799 
maintenance  actions.  There  was  no  correlation  found  between 
the  MTIP  con^letion  rate  and  the  DMMH/FH.  These  relationships 
have  provided  some  insight  with  encouraging  inplications  tor 
further  research  to  assess  the  existing  training  program. 
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I .  INTRODUCTION 


The  world  of  naval  aviation  has  been  in  existence  for  over 
eighty  years  since  Eugene  Ely,  a  civilian  pilot,  flew  a 
Curtiss  biplane  from  a  wooden  rcimp  built  on  the  forecastle  of 
USS  Birmingham  (Johnson,  1981) .  Since  then,  naval  aviation 
has  progressed  from  canvas  frcimed  aircraft  with  bicycle  tires 
and  wooden  propellers  to  highly  sophisticated,  complex 
airborne  weapons  systems.  The  complexity  of  modern  aircraft 
in  the  U.S.  Navy  requires  high  levels  of  knowledge  and  skill 
of  our  technicians  who  must  maintain  the  various  high-tech 
systems  so  that  the  aircraft  can  perform  their  intended  roles 
such  as  air-to-air  combat,  ordnance  delivery,  and  early 
airborne  warning,  just  to  mention  a  few. 

It  is  reasonable  to  believe  that  well -trained  personnel 
perform  better  on  their  jobs  than  do  less-well- trained 
personnel  (Gibson  and  Orlansky,  1986,  pp.5) .  If  this  is  so, 
the  effects  of  better  training  should  be  observable  in  such 
indicators  as  the  amount  of  direct  maintenance  manhours 
required  per  each  flight  hour  (DMMH/FH)  ,  the  number  of 
components  removed  by  the  Organizational  Maintenance  Level 
("0"  Level)  technicians  as  defective  that  are  found  to  operate 
properly  when  tested  later  (this  error  is  reported  as  a 
Malfunction  Code  A- 799) ,  and  the  operational  readiness 
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posture- -  Full  Mission  Capable  (FMC)  rate- -for  particular 
aviation  units.  If  the  effects  of  training  are  not  reflected 
in  unit  performance  and  operational  readiness,  then  the 
existing  training  progrcun  needs  to  be  either  improved  or 
replaced  by  an  alternative  program.  Deficiencies  in  technical 
training  and  personnel  qualifications  could  prevent  a 
technician  from  accjuiring  the  proper  knowledge  needed  to 
perform  at  his  or  her  top  efficiency.  The  purpose  of  aviation 
maintenance  training  is  to  prepare  sailors  to  perform  the 
technical  tasks  necessary  to  assure  the  availability  and 
proper  functioning  of  aircraft  weapons  systeirs  and  support 
equipment . 

A.  SCOPE  OP  STDDY 

In  any  naval  aircraft  community,  shortages  of  trained 
personnel  can  pose  a  serious  threat  to  naval  air  warfare 
operations.  The  Maintenance  Training  Improvement  Program 
(MTIP)  establishes  an  initial  level  of  proficiency  and 
provides  a  baseline  for  a  tailored  training  program  for 
improving  the  knowledge  of  technicians.  However,  the  program 
has  not  been  tested  to  determine  whether  it  is  ultimately 
helping  an  aviation  squadron  to  maintain  or  improve  its 
performance  at  a  competitive  level.  This  thesis  examines  the 
relationship  between  i;  ?IP  and  performance  indicators  such  as 
DMMH/FH,  Code  A- 799,  and  FMC  rates.  The  main  thrust  of  this 
study  is  to  determine  whether  there  is  any  correlation  between 
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MTIP  training  and  the  operational  readiness  of  the  Tomcat  (F- 
14)  and  Hawkeye  (E-2)  aircraft  communities  stationed  at  Naval 
Air  Station,  Miramar,  California. 

This  research  is  based  on  the  data  collected  from  the 
Flight  Activity,  Inventory  and  Readiness  Report;  the 
Individual  Maintenance  Action  Record;  and  the  Monthly  MTIP 
Completion  Rates  from  the  Fighter  Airborne  Early  Warning  Wing, 
U.S.  Pacific  Fleet  (FITAEWWINGPAC) . 

B.  THESIS  ORGANIZATION 

Chapter  II  discusses  the  evaluation  methods  available  to 
authors,  and  reviews  and  summarizes  the  previous  literature 
reporting  the  performance  measures  used  to  evaluate  the 
effectiveness  of  maintenance  training.  Examples  are  given  cf 
how  various  methods  of  measurement  were  performed. 

Chapter  III  provides  an  in-depth  background  of  the  Naval 
Aviation  Maintenance  Training  Program  (MTIP) .  Policies, 
procedures,  requirements,  minimum  standards,  and  assigned 
responsibilities  of  the  MTIP  training  are  discussed. 

Chapter  IV  presents  the  data  collected  from  the  Equipment 
Condition  Analysis  Report  and  the  MTIP  Completion  Rates.  Only 
relevant  information  from  this  raw  data  should  be  extracted, 
compiled,  and  displayed  in  a  logical  format.  Also  included  in 
this  chapter  are  the  analyses  that  describe  the  strength  of 
the  statistical  relationships  between  the  selected  performance 
indicators  and  MTIP  completion  rates. 
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Finally,  Chapter  V  contains  conclusions  and 
recommendations  about  the  impact  and  effectiveness  of  MTIP 
training  on  the  operational  readiness  of  aviation  squadrons. 
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II.  BACKGROUND 


A.  INTRODUCTION 

The  purpose  of  aviation  maintenance  training  is  to  prepare 
sailors  to  perform  the  technical  tasks  necessary  to  assure  the 
availability  and  proper  functioning  of  aircraft  weapons 
systems  and  support  equipment.  Aviation  maintenance  training 
programs  should  be  considered  dynamic  entities  that  slowly 
accomplish  their  purpose  in  meeting  predesigned  objectives. 
Without  systematic  evaluation,  there  is  no  feedback  to  provide 
the  information  necessary  to  improve  the  programs  through  the 
selection,  adoption,  and  modification  of  new  or  existing 
instructional  designs.  It  is  important  to  use  the  most 
systematic  procedures  available  that  fit  the  particular 
setting  being  investigated,  to  control  as  many  variables  as 
possible,  and  to  recognize  the  limitations  of  the  design  being 
used.  This  chapter  excimines  the  variables  that  can  affect  an 
evaluation  and  describes  the  various  evaluation  methodology 
available  to  researchers,  following  a  review  of  previous 
studies . 

B.  FACTORS  THAT  AFFECT  EVALUATION 

The  need  for  a  systematic  evaluation  of  the  Maintenance 
Training  Improvement  Program  (MTIP)  implies  a  need  for 
measurement  techniques  that  clearly  establish  the  relationship 
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between  training  and  performance  on-the-job.  In  training 
settings  that  can  be  tightly  controlled,  evaluation  can  be 
straightforward.  For  example,  on-site  training  in  a 
manufacturing  setting  is  conducive  to  evaluation. 
Conceivably,  personnel  could  be  tested  on  work  skills  before 
training  (pretesting) ,  training  would  be  completed,  and 
workers  would  be  tested  again  after  training  (posttesting) . 
In  such  a  setting  it  is  likely  that  job  performance  measures 
would  be  readily  available  and,  therefore,  post test 
performance  could  be  compared  to  pretest  performance  to 
evaluate  the  effectiveness  of  the  training.  Unfortunately, 
most  real-world  training  occurs  in  settings  where  such  tight 
control- -and  perfect  evaluation- -is  not  possible.  Two  areas 
that  make  training  evaluation  problematic  are  discussed  here: 
(1)  threats  to  internal  validity,  and  (2)  threats  to  external 
validity.  Threats  to  validity  can  be  controlled  through 
selection  of  the  appropriate  experimental  design,  which  is 
also  discussed  here. 

1.  Internal  Validity 

Internal  validity  refers  to  whether  the  training 
program  makes  a  difference  within  the  instructional  setting. 
That  is,  if  a  test  is  given  immediately  after  training,  e.g., 
in  the  classroom,  and  scores  are  higher  than  they  were  on  a 
test  given  immediately  before  training,  can  those  differences 
be  attributed  to  the  training- -did  learning  occur?  Of  course. 
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the  real  concern  is  with  transfer  of  learning  to  on-the-job 
performance,  that  is,  did  learning  occur  and  also  produce 
better  performance?  However,  the  issue  of  transfer  of 
training  cannot  be  dealt  with  unless  internal  validity  has 
been  established. 

Variables  other  than  instructional  events  that  affect 
the  results  of  training  are  threats  to  internal  validity. 
Examples  of  son.  of  these  variables  have  been  described  by 
Goldstein  (1993) : 

a.  History 

This  variable  refers  to  specific  events  occurring 
between  two  measurements  that  could  influence  or  provide 
alternative  explanations  for  the  results  of  a  training 
evaluation.  For  example,  with  the  outbreak,  of  the  Gulf  War, 
a  few  squadrons  were  required  to  take  the  posttest  without 
completing  the  entire  training  program  in  order  to  meet  the 
new  deployment  schedule.  This  unplanned  change  may  produce 
significant  differences  that  have  no  relationship  to  the 
material  presented  in  the  original  instructional  program. 

b.  Maturation 

This  refers  to  both  biological  or  psychological 
effects  that  vary  with  the  passage  of  time  (getting  older  or 
loss  of  interest  in  the  program  between  two  measurements)  and 
affect  performance  for  reasons  unrelated  to  a  training 
I  program. 
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c.  Testing 

A  pretest  may  influence  the  results  of  the 
posttest.  Simply  becoming  familiar  with  the  types  of 
questions  on  the  pretest  may  enable  a  maintenance  technician 
to  concentrate  on  materials  that  may  provide  answers  to  the 
next  test . 

d.  Inatrvmentation 

Changes  in  grading  standards,  configuration  changes 
with  a  new  model/type  of  equipment,  or  even  changing  the 
evaluator  can  result  in  differences  between  two  measurements 
that  make  it  difficult  to  determine  if  learning  occurred. 

e.  Selection  of  Participants 

It  can  be  difficult  to  determine  the  effects  of 
training  when  participants  bring  differing  amounts  of 
knowledge  to  the  learning  situation  that  cannot  be  accounted 
for  in  the  experimental  design.  For  example,  when  only  group 
performance  can  be  used  as  the  measure  of  training  even  though 
pretesting  and  posttesting  for  individuals  are  conducted 
(which  is  the  case  in  the  present  research)  ,  differences  could 
occur  betvjeen  the  experience  levels  of  different  groups. 
However,  this  variable  is  best  controlled  by  random  selection 
of  the  entire  population. 

f.  Interactions 

Any  of  the  variables  discussed  may  interact  to 
create  a  threat  to  internal  validity.  The  likelihood  of  this 
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occurring  is  increased  when  training  is  evaluated  outside  of 
a  controlled,  laboratory  setting. 

g.  Rivalry  between  Participants 

When  an  evaluation  is  made  public  or  when  two  units 
are  assigned  to  a  particular  condition,  the  special  effort 
based  on  motivation  may  wipe  out  a  true  reflection  of  how  each 
group  would  ordinarily  perform. 

2 .  External  Validity 

External  validity  deals  with  the  ultimate  concern  of 
training:  not  only  did  learning  occur,  but  can  it  be 

generalized  to  other  groups  and  settings?  Most  relevant  here 
is  the  issue  of  whether  training  is  related  to  on-the-job 
performance.  Threats  to  external  validity,  again  discussed  in 
Goldstein  (1993)  include  the  following: 

a.  Reactive  Effect  of  Pretesting 

As  noted  earlier,  a  pretest  can  sensitize 
individuals  to  information  presented  in  training  with  the 
results  of  improved  posttest  scores.  In  research  studies, 
where  evaluation  often  ta)  .:r  place,  pretesting  may  be  used 
during  the  research- -thereby  enhancing  the  effects  of 
experimental  training- -but  not  in  the  actual  implementation  of 
the  training. 
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b.  Interaction  of  Selection  and  Experimental 
Treatment 

The  characteristics  of  the  group  often  determine 
the  applicability  of  the  findings  for  other  purposes.  The 
characteristics  of  maintenance  technicians  from  one  squadron 
may  result  in  the  training  being  more  or  less  effective  for 
them  as  compared  to  an  another  squadron. 

c.  Reactive  Effects  of  Experimental  Settings 
Controlled  environments  with  special  setups  and 

equipment,  which  may  be  used  in  evaluation  research,  can  lead 
to  changes  in  behavior  that  will  not  be  present  when  training 
is  implemented  in  a  non- research  environment. 

3 .  Experimental  Design 

A  variety  of  experimental  designs  are  available  to 
evaluate  the  effects  of  training.  The  ideal  design  is  one 
that  would  produce  "clean"  results,  having  eliminated  all 
threats  to  validity.  With  such  an  ideal  model  one  could 
conclude  with  complete  confidence  that  observed  results  were 
a  function  of  the  training.  This  ideal  model  would  require 
the  availability  a  group  who  receives  training  (experimental 
group)  ,  matched  to  a  group  for  comparison  that  did  not  receive 
training  (control  group) ,  and  the  opportunity  for  pretesting 
and  posttesting  with  objective  performance  measures.  In  real- 
world  settings,  the  use  of  the  ideal  experimental  design  is 
rarely  possible.  Therefore,  the  experimental  design  is 
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tailored  to  accommodate  the  constraints  of  the  training  and/or 
on-the-job  setting.  These  modified  designs  will  control 
threats  to  validity  of  the  findings  differently.  While  not 
always  perfect,  many  researchers  have  discussed  the  importance 
of  doing  partial  evaluation  instead  of  avoiding  it  because  of 
the  inherent  difficulties  (Peterson,  1978) .  Some  of  the 
traditionally  used  designs  are  discussed  here. 

a.  The  One -group  Posttest  Only  Design 

Trainees  are  exposed  to  the  training  program 
without  a  pretest  and  then  are  tested  upon  completion  of 
training.  Without  the  pretest  baseline  and  a  control  group  to 
compare  it  with,  it  is  not  possible  to  ascertain  any  change  as 
a  result  of  the  training  program.  This  design  is  quite 
limited,  because  findings  cannot  be  generalized  to  other 
areas . 

Jb.  The  One -group  Pretest  and  Posttest  Design 

The  participants  are  given  a  pretest  before  the 
training  program,  and  a  posttest  upon  completion  of  training. 
This  design  provides  a  measure  of  comparison  between 
performance  before  and  after  the  training  program.  Again, 
without  a  control  group,  it  is  difficult  to  establish  whether 
the  change  is  due  to  the  training  program.  Internal  validity 
variables  such  as  history,  maturation,  testing  effects,  and 
changes  in  instrumentation  are  not  well  controlled. 


c.  Pretest  and  Posttest  Control -group  Design 

The  participants  are  chosen  at  random  and  assigned 
randomly  to  the  experimental  group  or  control  group.  Each 
group  is  given  a  pretest  and  posttest,  but  only  the 
experimental  group  is  provided  with  the  training  program. 
Many  of  the  internal  validity  threats  are  controlled. 
Variables  such  as  history,  maturation,  and  pretesting  affect 
both  groups  equally.  Differential  treatment  of  participants 
in  both  groups  must  still  be  controlled  by  the  evaluator. 
Pretesting  does  sensitize  and  affect  participants.  To  control 
this  sensitization,  a  third  group  may  be  introduced  without 
taking  the  pretest,  which  is  the  intent  of  the  next  design. 

d.  Solomon  Four-group  Design 

This  procedure  is  designed  to  consider  external 
validity  variables.  This  method  adds  two  groups  that  are  not 
pretested- -one  with  training,  one  without  training.  If  the 
participants  are  randomly  assigned  to  the  four  groups,  this 
design  makes  it  possible  to  compare  the  effects  of  pretesting. 
A  comparison  of  the  posttest  for  group  4,  which  was  not 
exposed  to  pretesting  or  training,  to  the  pretest  scores  of 
groups  1  and  2  makes  it  possible  to  analyze  the  combined 
effects  of  maturation  and  history. 

e.  The  Time -series  Design 

This  method  is  similar  to  the  one -group  post test 
design,  except  that  a  series  of  measurements  are  taken  before 
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and  after  training.  If  there  are  no  appreciable  changes 
through  successive  measurements,  it  is  unlikely  that  any 
effects  will  occur  due  to  maturation  or  testing.  The  major 
internal  validity  threat  with  this  method  is  the  history 
factor  where  events  may  change  as  environmental  changes  and 
historical  occurrences  are  not  controlled  by  this  procedure. 
This  design  does  not  control  most  of  the  external  validity 
threats.  Therefore,  it  is  necessary  to  be  sensitive  to 
factors  that  might  make  results  difficult  to  generalize  to 
other  groups . 

f.  The  Ndnequi valent  Control -group  Deaign 

This  design  is  similar  to  the  pretest  and  posttest 
control  group  design,  except  the  participants  are  not  assigned 
to  the  groups  at  random.  This  design  utilizes  groups  already 
established  such  as  class  cohorts,  where  manipulating 
assignment  to  groups  is  not  possible.  If  there  is  no 
alternative,  this  method  is  well  worth  using  and  is  preferable 
to  designs  that  do  not  have  control  groups  such  as  the  pretest 
and  posttest  group  design.  The  more  similar" the  two  groups 
and  their  scores  on  the  pretest,  the  more  effective  the 
control  becomes  in  accounting  for  factors  like  history, 
pretesting,  maturation,  and  instrumentation.  This  design  is 
vulnerable  to  interactions  between  maturation,  history,  and 
testing  because  the  participants  were  not  randomly  selected. 
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And  there  is  always  a  chance  that  they  differ  on  some  critical 
variable . 

4.  Other  Evaluation  Models 

While  variants  on  experimental  design  models  are 
traditionally  used  for  evaluating  training,  there  are  other 
approaches  that  can  be  used. 

a.  Individual -differeace  Model 

This  method  uses  statistical  methods  to  measure  the 
relationship  between  training  and  actual  job  performance.  The 
emphasis  here  is  on  demonstrating  that  performance  in  training 
will  predict  performance  on  the  job.  Typically,  correlation 
coefficients  are  used  to  measure  the  relationships  of 
interest.  The  problem,  of  course,  is  that  correlations 
measure  relation,  not  causation.  A  strong  correlation  in  this 
situation  means  that  people  who  do  well  in  training  do  well  on 
the  job.  The  correlation  does  not  say  that  the  training 
program  caused  people  to  do  well  on  the  job.  This  concept  is 
relevant  to  this  thesis  and  will  be  discussed  further  in  the 
discussion  of  the  results. 

b.  Content- validity  Model 

This  method  relies  on  the  assumption  that 
judgements  can  be  made  about  the  adequacy  of  the  design  of  the 
training  program.  If  the  needs  assessment  has  been  done 
correctly  and  has  been  used  to  determine  the  knowledge, 
skills,  and  abilities  (KSAs)  taught  in  the  program,  then  the 


program  is  content  valid.  Effectively,  the  concept  is  that  a 
good  needs  analysis  automatically  guarantees  that  the  training 
is  related  to  the  KSAs  needed  from  the  job.  The  problem  is 
that  the  method  does  not  address  whether  the  KSAs  have  been 
learned  or,  in  fact,  transferred  to  the  job.  Assuring  content 
validity  and  performing  correlational  analyses  would  provide 
a  more  adequate  means  of  evaluating  training. 

C.  REVIEW  OP  PREVIOUS  STUDIES 

There  has  never  been  a  study  conducted  to  evaluate  the 
Maintenance  Training  Improvement  Program  (MTIP) .  However,  a 
few  studies  have  examined  operational  readiness  measures  such 
as  malfunction  code  A- 799  no-defect  rate,  sortie  completion 
rate,  and  elapsed  maintenance  time.  These  studies  demonstrate 
some  of  the  approaches  that  have  been  taken  to  evaluate 
training  in  real-world  settings.  Examples  of  these  are 
suiTjnarized  below: 

1.  Orlansky  and  Striiig  (1981)  ,  reviewed  a  group  of  seven 
studies  across  all  military  services.  The  evaluation  method 
used  in  this  review  does  not  fit  into  any  models  described  in 
this  chapter.  Basically,  information  was  collected  and  found 
that  non- faulty  parts  were  removed  in  4  to  43  percent  of  all 
corrective  maintenance  actions  accounting  for  9  to  32  percent 
of  all  maintenance  man-hours.  Non- faulty  parts  are  those  that 
were  removed  but  found  not  to  be  defective  when  received  for 
repair  by  the  next  level  of  maintenance  activity.  The 


researchers  concluded  that  these  data  offer  strong  evidence 
that  maintenance  technicians  may  conduct  maintenance  in  an 
inappropriate  and  inefficient  manner.  The  researchers  offer 
no  inference  to  identify  the  factors  that  may  lead  to 
inappropriate  maintenance.  However,  training  is  considered  a 
possible  factor  contributing  to  the  high  A- 799  rate. 

2.  Horowitz  and  Angier  (1985)  found  that  the  experience 
level,  as  indicated  by  the  paygrades  of  the  maintenance 
personnel,  is  the  most  consistent  predictor  of  readiness. 
Using  the  number  of  Navy  A- 7  aircraft  flights  off  a  carrier  in 
a  quarter  as  a  measure,  a  positive  relation  between  paygrades 
and  the  number  of  sorties  completed  per  quarter  was  observed. 
The  researchers  concluded  that  formal  training  and  on-the-job 
experience/learning  have  an  observable  and  meaningful  impact 
on  operational  measures.  However,  the  evaluation  design  used 
in  this  study  again  does  not  resemble  any  of  the  models 
described  in  this  chapter.  No  pretest  or  posttest  nor  control 
group  was  employed  in  this  study.  Again,  a  relationship  was 
measured- -not  causation- -and  the  results  could  have  been 
caused  by  variables  such  as  history,  maturation,  or 
motivation.  Without  controlling  these  factors,  ascertaining 
any  change  in  the  performance  as  a  result  of  training  is  not 
possible . 

3.  Johnson,  McConnell,  and  Murdock  (1983),  used  data  from 
the  Air  Force  Consolidated  Data  System  for  F-16  aircraft  and 
found  that  speed  in  accomplishing  maintenance  tasks  was 
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related  to  the  completion  of  formal  school  training  provided 
by  the  Field  Training  Detachment  (FTD)  .  No  pretest  or 
posttest  was  conducted  to  measure  the  effects  before  and  after 
the  training  program.  Variables  such  as  history  and 
maturation  were  not  positively  controlled.  However,  control 
groups  were  used  to  evaluate  the  effects  of  formal  training. 
Maintenance  work  centers  with  over  60  percent  FTD- trained 
personnel  were  compared  with  work  centers  with  less  than  60 
percent  FTD- trained  personnel.  The  data  collected  were 
limited  to  Work  Unit  Codes  that  are  used  co  identify  the 
system,  subsystem,  assembly  and  component,  etc.,  on  which  the 
maintenance  task  is  performed.  Formal  training  had  a  greater 
effect  on  reducing  the  time  needed  to  perform  maintenance  than 
did  the  number  of  maintenance  actions  performed  by  a  worker. 
The  study  finds  that  work  hours  used  to  complete  a  task  is  a 
meaningful  and  useful  productivity  measure  that  is  sensitive 
to  differences  in  training  backgrounds  and  methods. 

As  evidenced  in  real-world  settings  of  these  studies, 
there  are  no  proven  off-the-shelf  methodologies  for  evaluating 
maintenance  training  programs  where  both  internal  and  external 
threats  can  be  tightly  controlled.  The  primary  goals  of  the 
Maintenance  Training  Improvement  Program  (MTIP)  are  to  raise 
technical  know] edge  levels,  to  contribute  to  unit  performance 
and  to  enhance  and  imp...ove  existing  formal  training.  Without 
systematic  evaluation,  there  is  no  feedback  to  provide 
information  necessary  to  improve  this  program. 
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The  next  chapter  examines  the  Maintenance  Training 
Improvement  Program  in  detail.  Policies,  procedures, 
requirements,  minimum  standards,  and  assigned  responsibilities 
of  the  MTIP  training  are  discussed.  The  methodology  for  this 
thesis  study  relative  to  the  concepts  discussed  here  are 
explained  in  Chapter  IV. 
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III.  NAINTENANCE  TRAINING  IMPROVEMENT  PROGRAM 


A.  INTRODUCTION 

The  Assistant  Chief  of  Naval  Operations  (Air  Warfare) ,  as 
program  sponsor,  provides  the  following  overall  program 
direction: 

The  Maintenance  Training  Improvement  Program  (MTIP)  is 
a  training  management  system  which  shall  be  implemented 
throughout  naval  aviation  (Chief  of  Naval  Operations, 
OPNAV  Instmction  4790. 2E,  June  1990). 

Since  the  mid- 80s,  MTIP  has  become  an  integral  part  of  the 
overall  aviation  maintenance  training  program.  It  is 
compatible  with  and  supports  the  formal  technical  training 
programs  in  naval  aviation.  MTIP  is  designed  to  supplement 
formal  training  provided  by  the  Naval  Air  Maintenance  Training 
Group  (NAMTD)  and  Fleet  Readiness  Aviation  Maintenance 
Personnel  (FRAMP) .  All  personnel  should  receive  formal 
training  prior  to,  or  upon  assignment  to,  a  maintenance 
production  work  center.  In  cases  where  formal  training  is  not 
received,  MTIP  testing  is  the  only  assessment  tool  available 
to  determine  an  individual's  knowledge  level.  The  primary 
goals  are  to  raise  technical  knowledge  levels,  and  to  enhance 
and  improve  existing  formal  training. 
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The  Maintenance  Training  Improvement  Program  (MTIP)  is  the 
primary  method  used  to  identify  personnel  training 
deficiencies  at  the  organizational  and  intermediate 
maintenance  levels.  Through  diagnostic  testing  procedures 
consisting  of  standardized  question  and  answer  banks  from 
which  computer  generated  tests  are  assembled,  a  qualitative 
assessment  of  individual  technical  knowledge  can  be  made.  The 
questions  are  designed  to  test  knowledge  of  systems  for  each 
occupational  specialty/rating  required  of  a  maintenance 
technician  on  the  job.  As  deficiencies  are  identified, 
refresher  or  remedial  training  is  conducted.  Such  assessments 
not  only  help  to  improve  existing  training  materials  and 
courses  but  also: 

1.  Establish  the  baseline  knowledge  level  of  an  aviation 
community  such  as  F-14  TOMCAT  or  E-2  HAWKEYE,  or  a  particular 
squadron,  work  center,  and/or  an  individual  technician 
compared  to  his  or  her  peers. 

2.  Concentrate  training  efforts  where  they  are  most 
needed,  according  to  the  established  baseline". 

3.  Provide  a  training  program  tailored  to  individuals. 

4.  Correct  specific  systems  knowledge  deficiencies  prior 
to  extended  deployment  while  dedicated  training  resources  are 
available . 

5.  Provide  a  feedback  loop  across  all  phases  of  training, 
and  interface  with  all  echelons  from  the  user  through  the 
functional  wing  (COMFITAEWWINGPACINST  1540. IJ). 
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MTIP  training  primarily  relies  on  static  media  such  as 
printed  technical  manuals,  workbooks,  and  slides.  There  is 
also  some  use  of  films  and  video,  but  the  use  of  advanced 
instructional  technologies  (e.g.,  computer-based  training)  has 
not  been  incorporated.  MTIP  provides  units  with  some 
flexibility  to  direct  their  own  inservice  training  program  and 
yet  be  able  to  compare  the  outcomes  to  established  standards. 
However,  as  the  rate  at  which  a  squadron  operates  increases, 
the  progrcim  is  often  seen  as  administratively  burdensome  and 
secondary  to  operational  requirements  (Ledeboer,  1988) .  The 
operational  tempo  and  commitments  often  become  the  driving 
factors  for  accomplishing  the  program. 

MTIP  is  a  training  management  system  used  to  identify 
aviation  personnel  training  deficiencies  throughout  the  Navy, 
but  the  ultimate  goal  of  MTIP  is  to  improve  the  operational 
readiness  of  aviation  squadrons.  It  is  essential  that 
training  of  aviation  maintenance  personnel  be  afforded  the 
same  level  of  attention  as  flight  crew  training.  An  effective 
MTIP  requires  high  emphasis  and  dedicated  effort  at  every 
level  of  command  (COMFITAEWWINPACINST  1540. IJ).  This  thesis 
examines  whether  there  is  any  relationship  between  MTIP  and 
the  operational  readiness  of  F-14  and  E-2  squadrons  within  the 
Fighter  Airborne  Early  Warning  Wing,  Pacific  located  at  Naval 
Air  Station,  Miramar. 
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B.  TEST  REQUIREMENTS 

All  sea  and  shore  duty  aviation  personnel  (E-6  and  below)  , 
including  staff  and  administrative  support  services  such  as 
Quality  Assurance  Representatives  (QAR) ,  Aviation 
Supply/Storekeepers  (AK)  ,  and  Aviation  Administrationman  (AZ) 
who  are  assigned  to  an  aviation  maintenance  department  work 
center,  are  administered  MTIP  testing.  QARs  consist  of 
senior,  highly  experienced  technicians  from  various  work 
centers  (e.g..  Power  Plants,  Avionics,  etc.).  They  are  hand¬ 
picked  by  their  divisions,  qualified  by  the  Quality  Assurance 
Division,  and  officially  designated  by  the  Maintenance  Officer 
in  accordance  with  the  requirements  set  forth  in  OPNAVINST 
4790. 2E.  They  are  detached  from  their  work  center  and 
assigned  to  serve  as  staff  members  responsible  to  the  Quality 
Assurance  Division.  All  QARs  are  tested  not  only  in  the 
technical  knowledge  of  their  occupational  specialty  but  also 
for  skills  in  quality  assurance  administration.  All  E-5s  and 
E-6s  assigned  to  production  work  centers  in  supervisory 
positions  are  tested  in  their  occupational  specialty,  as  well 
as  on  Naval  Aviation  ?'Iai.atenance  Program  (NAMP)  supervisory 
knowledge.  Tests  on  NAMP  supervisory  knowledge  are  not 
required  of  QARs,  AKs,  or  AZs. 
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1.  Sea  Duty  Persozmel 

For  aviation  squadrons  deploying  as  part  of  a  Carrier 
Air  Wing  Group  (CVW) ,  MTIP  testing  is  scheduled  by  the  parent 
functional  wing  Maintenance  Training  Team.  There  are  two 
required  tests  for  each  complete  training  cycle  (from 
completion  of  a  deployment  to  the  start  of  a  next  deployment)  , 
which  varies  in  length  in  accordance  with  the  Commander  Naval 
Air  Force  United  States  Pacific  Fleet  (CNAP)  deployment 
schedule  and  averages  one  to  one  and  half  years.  The  two 
required  tests  are  discussed  in  the  following  paragraphs: 

a.  Post-d^loymeat  Test  (CM) 

Testing  materials  are  sent  by  the  functional  wing 
Maintenance  Training  Team  to  the  Carrier  Air  Wing  Group  (CAG) 
Maintenance  Officer  for  administration  of  post -deployment 
testing  by  the  squadron  Assistant  Maintenance  Officer.  Those 
personnel  who  will  not  take  part  in  the  next  deployment  are 
not  required  to  take  the  CM  tests.  However,  any  prudent 
maintenance  officer  will  undoubtedly  strongly  encourage  all 
personnel  to  take  the  post -deployment  test  Tor  the  sake  of 
providing  continuous  training  until  an  individual  is  ready  for 
transfer.  CM  tests  may  be  completed  up  to  forty- five  days 
prior  to  completion  of  the  deployment,  but  no  later  than 
thirty  days  after  return  from  deployment  (COMFITAEWWINGPACINST 
1540. IJ).  Completing  the  post -deployment  test  early  enables 
the  packages  to  be  graded,  analyzed,  where  deficiencies  are 


noted  and  compared,  and  presented  to  the  Commanding  Officer  of 
each  squadron  at  the  earliest  possible  time.  An  aviation 
unit's  turnaround  (from  completion  of  a  deployment  to  the 
start  of  a  next  deployment)  Maintenance  Training  Improvement 
Program  could  commence  immediately  upon  receiving  the  insults. 
The  various  MTIP  reports  provided  initially  by  the  Maintenance 
Training  Team  (MTT)  help  the  squadron  maintenance  training 
officer  or  chief  petty  officer  to  manage  the  training  program 
by  tailoring  each  individual's  needs  in  accordance  with 
his (her)  areas  of  deficiencies.  Whenever  an  updated  status  of 
the  training  requirements  is  desired,  MTIP  reports  may  be 
requested  via  the  Aviation  Training  Support  System  (ATSS) 
computer  terminal  located  at  each  squadron. 

b.  Pre- deployment  Teat  (CP) 

CP  tests  must  be  completed  no  later  than  ninety 
days  prior  to  a  deployment.  All  personnel  (E-6  and  below)  who 
are  ta)cing  part  in  the  deployment  are  administered  a  pre- 
deployment  test . 

2 .  Shore  Duty  Personnel 

For  those  aviation  maintenance  personnel  (E-6  and 
below)  assigned  to  units  that  do  not  deploy  as  a  part  of  a 
Carrier  Air  Wing  Group  on  a  regular  basis  (e.g.,  training 
commands  such  as  VF-124  and  VAW-llO  squadrons  which  are 
responsible  for  training  maintenance  personnel  for  F-14  and  E- 
2  aircraft  respectively) ,  MTIP  testing  is  completed  annually. 
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Those  who  have  less  than  six  months  remaining  on  a  tour  need 
not  be  tested.  But  again,  for  the  benefit  of  an  individual's 
professional  growth,  especially  if  his  or  her  next  assignment 
will  involve  a  similar  type  of  aircraft,  the  squadron 
maintenance  officer  should  continue  to  identify  weaknesses 
and  to  improve  technical  knowledge  and  skill . 

3.  Newly  Assigned  Personnel 

Newly  assigned  personnel  (E-3  and  below)  who  have  not 
completed  Fleet  Readiness  Aviation  Maintenance  Personnel 
(FRAMP)  pipe -line  training  are  tested  within  six  months  of 
reporting  to  a  maintenance  production  work  center.  This  time 
requirement  is  not  applicable  to  personnel  assigned  Temporary 
Duty  (TAD)  as  a  barracks  compartment  cleaner  or  as  a  member  of 
the  First  Lieutenant  Division  responsible  for  cleanliness  of 
squadron  spaces.  Temporary  Assigned  Duty  may  last  an  average 
of  three  months  or  more  depending  on  the  shortages  of  junior 
personnel.  Therefore,  it  is  possible  that  these  personnel 
will  not  become  part  of  the  Maintenance  Training  Improvement 
Program  until  nine  or  more  months  after  checking  into  a 
squadron . 

All  other  newly  assigned  personnel  are  tested  within 
thirty  days  of  assignment  to  a  maintenance  department  work 
center.  If  an  individual  has  completed  FRAMP  completion  tests 
(CF)  within  30  days  of  reporting  to  a  work  center,  the  test 
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results  may  be  used  to  satisfy  the  initial  MTIP  testing 
requirements . 

C.  TASK  AREAS  TO  BE  TESTED 

The  composition  and  number  of  work  centers  vary  according 
to  the  type  of  aircraft  an  activity  maintains  and  operates. 
For  excunple,  Radar  Fire  Control  Technicians  (AQ)  and  Aviation 
Ordnance  Technicians  (AO)  are  not  needed  in  an  E-2  squadron. 
Even  though  work  centers  such  as  Aviation  Administration, 
Aviation  Storekeepers,  Quality  Assurance,  Power  Plants, 
Airframes,  Parachute  Riggers,  Environmental  Control, 
Electronics,  Electric,  and  Plane  Captain/Trouble  Shooters  are 
common  to  both  F-14  and  E-2  squadrons,  system  knowledge  and 
skills  required  of  the  respective  technicians  differ 
considerably  from  one  aircraft  to  another.  Therefore,  MTIP 
tests  and  lesson  guides  are  tailored  accordingly  to  meet  an 
activity's  requirements.  They  are  based  on  applicable 
technical  manuals,  work  unit  codes,  the  Naval  Aviation 
Maintenance  Program  (OPNAVINST  4790. 2E),  and  other 
instructions.  All  test  questions  and  lesson  guides  have  a 
valid  reference  or  training  source  sailors  can  study. 

Areas  of  knowledge  required  of  a  technician  can  be 
niimerous.  For  example,  an  Airframe  Technician  is  required  to 
be  familiar  with  task  areas  in  the  landing  gear  system, 
hydraulics  system,  aileron  system,  rudder  system,  brake 
system,  and  pneumatic  system.  Each  task  area  for  each 
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occupational  specialty  is  included  in  the  MTIP  test.  An  E-2 
squadron  alone  consists  of  approximately  121  task  areas  to  be 
tested  and  managed.  Every  task  area  must  meet  the  minimum 
standard  set  forth  by  the  functional  wing  Maintenance  Training 
Team.  Depending  on  the  skill  level,  past  training,  and 
experience  level  at  the  time  of  MTIP  testing,  it  is  possible 
for  a  squadron  to  accumulate  three  to  four  hundred  combined 
deficiencies  relative  to  the  minimum  standards. 

D.  REFRESHER  TRAINING  REQUIREMENTS 

1.  Minimum  Test  Scores 

MTIP  remedial  training  is  required  for  all  test  areas 
in  which  scores  are  less  than  the  previous  year's  functional 
wing  average  or  sixty- three  percent  (whichever  is  higher) . 
Depending  on  the  task  area,  the  minimum  cut-off  average  score 
may  be  set  as  high  as  seventy- five  percent  by  the  functional 
wing.  Those  individuals  who  do  not  meet  the  cut-off  grades 
are  considered  deficient  in  those  task  areas.  Refresher 
training  is  also  available  to  those  who  have  successfully 
attained  scores  above  the  minimum  standard  but  wish  to  improve 
or  maintain  their  knowledge  of  the  latest  information. 

2.  Types  of  Refresher  Training 
a.  Scheduled  Training 

Formal  MTIP  refresher  training  is  the  primary 
method  used  to  correct  documented  deficiencies.  Subject 
Matter  Experts  (SME)  and  other  highly  experienced  technicians 


from  Fleet  Readiness  Aviation  Maintenance  Personnel  (FRAMP) , 
Naval  Aviation  Engineering  Services  Unit  (NAESU) ,  and  Naval 
Air  Maintenance  Training  Group  (NAMTD)  are  designated  as 
training  instructors.  Quarterly  MTIP  training  classes  are 
scheduled  and  published  by  the  functional  wing  Maintenance 
Training  Team.  Quotas  for  training  classes  are  assigned  as 
requested  by  squadrons .  Due  to  the  large  number  of  task  areas 
to  be  covered,  a  particular  training  class  may  or  may  not  be 
repeated  within  the  saune  quarter.  But  over  time,  every  task 
area  will  be  covered  as  many  times  as  possible  to  satisfy 
every  squadron's  needs. 

Upon  completion  of  refresher  training,  the  MTT  ensures 
retesting  is  conducted  for  each  individual  as  soon  as  the 
class  is  completed.  Additional  refresher  training  is  required 
if  the  minimum  standard  is  not  met. 
b.  In-houae  Training 

MTIP  lesson  guides  similar  to  those  used  by  the 
designated  instructors  are  available  in  computer  disks  and 
distributed  to  each  squadron  for  conducting  refresher  training 
at  their  own  convenience.  Reviews  and  changes  to  these  lesson 
guides  are  coordinated  by  the  MTT.  Subject  matter  experts 
from  FRAMP,  NAESU,  NAMTD  and  the  squadrons  play  a  vital  role 
in  keeping  these  lesson  guides  accurate  and  current.  Using 
these  lesson  guides,  training  can  be  conducted  anywhere  and 
anyplace  that  operational  tempo  permits,  regardless  of  whether 
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a  squadron  is  on  an  aircraft  carrier  for  work-ups  or  on  a 
detachment  to  a  remote  training  ground.  Training  c[uality  with 
this  method  can  be  comparable  or  better  than  the  formal 
refresher  training  at  the  home  base  if  the  subject  matter 
experts  from  NAESU  or  the  contractors  accompany  the  squadron 
as  a  member  of  the  detachment-  Retesting  can  be  administered 
on  site  or  upon  returning  to  the  home  base  but  tests  can  only 
be  graded  by  the  functional  wing  Maintenance  Training  Team. 
Testing  materials  are  treated  with  the  utmost  security  and 
confidentiality. 

c.  Integrated  Wieapons  Systems  Review  fJMSRj 

One  other  very  useful  and  effective  method  of 
hands-on  training,  mostly  applicable  to  the  Electrical  and 
Avionics  ratings,  is  the  Integrated  Weapons  System  Review 
(IWSR) .  With  squadron  aircraft  assigned  strictly  for 
maintenance  training,  maintenance  technicians  can  proceed 
through  the  learning  process  with  the  help  of  subject  matter 
experts  from  NAESU  or  the  contractor.  Due  to  the  involvement 
of  operational  aircraft  along  with  outside  technical  experts, 
IWSR  requires  tremendous  planning  and  scheduling  effort. 
Therefore,  only  the  more  experienced  technicians  are  normally 
assigned  to  this  type  of  training.  Assigned  aircraft  and 
technicians  involved  in  IWSR  are  not  available  for  the  day-to- 
day  operational  commitments  of  a  scjuadron  which  may  last  as 
long  as  two  to  three  weeks.  IWSR  is  normally  scheduled 


towards  the  latter  part  of  a  training  cycle.  IWSR  may  be 
considered  the  climax  of  turnaround  {from  completion  of  a 
deployment  to  the  start  of  a  next  deployment)  training  effort, 
signaling  the  nearness  of  an  upcoming  extended  deployment. 
The  experience  gained  by  these  few  assigned  technicians  during 
the  IWSR  is  expected  to  be  passed  on  to  the  rest  of  work 
centers . 

3 .  MTIP  Co]q)letlon  Rates 

Pre- deployment  tests  for  sea  duty  personnel  and  annual 
tests  for  shore  duty  personnel  mark  the  end  of  a  training 
cycle  and  freeze  the  percentage  score  of  an  aviation  unit.  The 
MTIP  completion  rate  is  t  le  number  of  deficiencies  corrected 
divided  by  the  total  number  of  CM  and  CF  deficiencies  times 
100.  For  excimple,  if  a  squadron  completes  293  of  the  combined 
CM  and  CF  total  of  470  deficiencies  during  a  turnaround 
training  cycle,  it  is  said  to  have  attained  a  62  percent  MTIP 
completion  rate.  A  minimum  of  seventy- five  percent  of 
deficiencies  noted  in  the  post -deployment  (CM)  and  FRAMP 
completion  (CF)  tests  is  to  be  corrected  using  scheduled 
training  and/or  in-house  training  (both  described  earlier  in 
this  chapter)  during  the  turnaround  training  cycle. 

4.  MTIP  On- the- Job  Training  (OJT) 

On- the- Job  (OJT)  is  the  practical  instruction  of 
personnel  in  the  performance  of  maintenance  tasks.  Under  the 
supervision  of  qualified  personnel,  the  trainee  learns  to 
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complete  tasks  using  an  appropriate  maintenance  manual. 
Experienced  personnel  are  used  to  instruct,  demonstrate  and 
impart  their  skills  to  the  less  experienced.  Trainees  learn 
by  observing  job  performance  and  gain  experience  by 
participating  in  the  work  effort.  OJT  syllabi  are  prepared  to 
complement  MTIP  task  areas.  Attainment  of  a  task  area  score 
at  or  above  the  cut-off  on  either  the  MTIP  test  or  refresher 
training  test  is  a  prerequisite  to  final  MTIP  OJT 
qualification  in  a  task  area.  When  an  individual  completes 
MTIP  OJT  qualification,  he  (she)  is  considered  fully  q'aalified 
to  perform  the  task  indicated.  This  qualification  will  remain 
in  force  until: 

1,  A  score,  on  a  subsequent  CM  MTIP  test,  indicates  a 
requirement  for  refresher  training  on  those  task  areas  with 
scores  below  the  cut-off. 

2.  Transfer  to  an  another  unit. 

E.  MTIP  TRAINING  REPORTS 

Aviation  Training  Support  System  (ATSS)  is  a  Manpower, 
Personnel  and  Training  (MPT)  Automatic  Data  Processing  (ADP) 
system  that  provides  principal  support  for  FRAMP,  NAMTD, 
Aircraft  Intermediate  Maintenance  Department,  and  operational 
fleet  squadrons  during  the  turnaround  training  cycle 
(COMFITAEWWINGPACINST  4500.3). 

ATSS  is  used  for  MTIP  data  collection,  testing  and 
reporting.  Access  to  the  MTIP  data  base  is  controlled  by  user 
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identification  codes  and  passwords.  Currently,  ATSS  has 
seventeen  separate  reports  (see  Appendix  A)  available  under 
the  Maintenance  Training  Improvement  Program.  Most  of  these 
reports,  especially  those  involving  test  question  data,  are 
not  available  to  an  aviation  unit.  The  most  commonly  used 
reports  directly  accessible  by  a  squadron  are: 

MOl  MTIP  Personnel  Training  Report 

M02  MTIP  Work  Center/System  Training 

M05  MTIP  Squadron/Work  Center  CDP  Report 

MIO  MTIP  Squadron  Training  Requirements  Reports 

M21  Squadron  Turnaround  Training  Progress  Report 
With  these  reports,  a  squadron  training  officer  or  chief  petty 
officer  has  all  the  information  needed  to  concentrate  training 
effort  where  it  is  most  needed.  He  or  she  has  the  option  to 
go  beyond  the  MTIP  refresher  training  by  sending  those 
individuals  back  to  formal  schools  offered  by  other  training 
commands  such  as  NAMTD. 

F.  SUMMARY 

MTIP  has  become  an  integral  part  of  the  overall  aviation 
maintenance  training  program  in  the  Navy.  It  is  a  training 
management  system  used  to  identify  training  deficiencies  at 
the  various  levels,  ranging  from  an  individual  to  the  aviation 
community  as  a  whole.  Through  diagnostic  testing  procedures, 
a  qualitative  assessment  of  individual  technical  knowledge  is 
made  for  all  aviation  maintenance  personnel.  Based  on  this 
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information,  the  training  program  is  tailored  to  an 
individual's  needs  and  concentrates  efforts  where  they  are 
most  needed.  The  completion  rates  for  each  aviation  squadron 
are  used  to  measure  the  overall  effectiveness  of  Maintenance 
Training  Improvement  Program.  However,  the  effects  of  a 
training  program  should  reflect  on  unit  performance  and 
operational  readiness,  which  provide  the  final  measure  of 
effectiveness  and  represent  the  end  products  of  the 
maintenance  training  system. 

The  next  chapter  examines  the  effects  of  MTIP  on  F-14  and 
E-2  squadrons  at  Naval  Air  Station,  Miramar,  California. 
Using  the  MTIP  completion  rates  and  information  collected 
under  the  Naval  Aviation  Logistics  Data  Analysis  (NALDA) , 
analyses  are  conducted  on  the  relationships  between  training 
and  operational  readiness. 
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IV.  METHOD  AMD  RESULTS 

A.  INTRODUCTION 

This  chapter  discusses  the  method,  participants, 
materials,  procedures,  and  limitations  of  this  study.  It  also 
presents  the  data  that  are  used  in  analyzing  relationships 
between  Maintenance  Training  Improvement  Program  (MTIP) 
completion  rate  and  operational  readiness  measures  such  as 
Full  Mission  Capable  (FMC)  rate,  number  of  no-defect  (A- 799) 
maintenance  actions,  and  Direct  Maintenance  Manhours  per 
Flight  Hour  (DMMH/FH) . 

B.  METHOD 

Training  is  the  systematic  acquisition  of  s)cills,  rules, 
concepts,  or  attitudes  that  result  in  improved  performance  in 
another  environment  (Goldstein,  1993) .  In  order  to  capture 
these  two  environments,  a  measure  comparing  performance  before 
training  and  an  assessment  of  on-the-job  performance  following 
training  are  needed  to  evaluate  the  effects  of  the  Maintenance 
Training  Improvement  Program. 

As  discussed  in  Chapter  II,  a  variety  of  experimental 
designs  are  available  and  of  course,  the  ideal  model  is  one 
that  would  produce  "clean"  results,  having  eliminated  all 
threats  to  validity.  But  most  real-world  training  occurs  in 
settings  where  perfect  evaluation  with  an  ideal  model  is 
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rarely  possible.  Due  to  the  inherent  nature  of  military 
service,  with  the  necessity  to  undertake  operations  on  short¬ 
term  notice,  an  aviation  squadron  may  be  called  into  action 
thousands  of  miles  away  from  home  base  for  an  extended  period 
of  time.  It  is  not  practical  to  collect  data  under  test 
conditions  that  need  to  be  designed  and  set  up  each  time 
information  is  needed.  It  is  logical  to  seek  a  methodology 
based  on  practicality  and  on  existing  available  information  to 
track  long-term  trends  in  unit  performance  and  combat 
readiness . 

Therefore,  the  experimental  design  needs  to  be  tailored  to 
accommodate  the  constraints  of  the  training  and/or  on-the-job 
setting.  A  desirable  design  would  differentially  control  as 
many  threats  to  internal  and  external  validity  as  possible, 
and  minimize  constraints  imposed  by  the  environment  and  the 
influences  of  the  multitude  of  organizational  variables. 
While  not  perfect,  the  data  collected  under  these  conditions 
makes  it  possible  to  avoid  a  useless  evaluation.  As  Peterson 
(1978)  suggests. 

It  is  better  to  approximate  evaluation  than  neglect  it 
altogether  due  to  its  difficulties.  For  example, 
measuring  learning  not  only  at  the  end  of  the  course,  but 
sometime  later  is  better  than  nothing  just  because  you 
can't  get  measures  of  behaviors  or  results. 

Based  on  the  concepts  discussed  in  Chapter  II,  and  to 
accommodate  a  real-world,  on-the-job  setting  as  explained 
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atbove,  the  locally  availcdale  MTIP  records  combined  with  the 
Navy's  existing  maintenance  data  base  from  Naval  Aviation 
Logistics  Data  Analysis  (NALDA)  were  selected  for  this  study. 
These  data  provide  a  practical  approach  to  measurement,  and 
also  reflect  the  ultimate  outcome  that  one  would  expect  from 
a  training  program. 

MTIP  is  conducive  to  the  one-group  pretest/posttest 
evaluation  method.  All  maintenance  personnel  (like  an 
experimental  group)  assigned  to  each  squadron  are  given  a 
pretest  (post -deployment  test)  to  estadilish  the  baseline  of 
training  deficiencies,  then  presented  with  the  training 
program,  and  finally  given  a  posttest  (pre- deployment  test) . 
The  MTIP  con^letion  rate  (Chapter  III,  section  D,  subsection 
3)  represents  the  final  effort  of  the  training  program  in 
terms  of  percentage  completed  during  the  allotted  time  period. 
Appendix  B  lists  all  squadrons'  completion  rates  for  this 
study.  Newly  assigned  personnel  and  those  scheduled  to  be 
transferred  before  the  next  deployment  are  automatically 
excluded  from  the  MTIP  completion  rate  calculation  by  the 
Aviation  Training  Support  System  (ATSS)  computer  data 
management  program.  This  method  provides  a  measure  of 
comparison  between  before  and  after  training  for  the  Scune 
group  of  personnel.  Unfortunately,  without  permission  from 
the  highest  level  of  authority  in  the  Navy,  and  also  for 
obvious  practical  reasons,  control  groups  are  not  feasible  for 
the  purpose  of  this  evaluation  process.  Therefore,  it  is 
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difficult,  without  a  control  group,  to  ascertain  that  the 
training  program  is  the  prime  factor  determining  any 
differences  that  occur  between  measurements. 

On  the  other  hand,  the  Naval  Aviation  Logistics  Data 
Analysis  (NALDA)  System  provides  a  data  base  for  fleet 
maintenance  and  operations.  It  provides  information  on  unit 
performance  and  operational  readiness  of  an  aviation  unit. 
Full  Mission  Capable  (FMC)  rate,  number  of  no-defect  (A- 799) 
maintenance  actions,  and  Direct  Maintenance  Manhours  per 
Flight  Hour  (DMMH/FH)  were  extracted  from  the  NALDA  system; 
they  are  listed  in  Appendices  C  and  D.  The  FMC  rate  is  a 
percentage  figure  based  on  the  monthly  total  available 
equipment  in-service  hours  that  all  systems  are  fully 
functional  to  perform  all  missions  designed  for  an  aircraft. 
A- 799  actions  are  the  number  of  maintenance  actions  involving 
components  removed  by  the  Organizational  Maintenance  Level 
("0"  Level)  technicians  as  defective  that  are  found  to  operate 
properly  when  tested  later.  DMMH/FH  is  the  average  amount  of 
direct  maintenance  manhours  required  to  support  each  flight 
hour.  These  operational  measures  reflect  what  should  be 
learned  as  a  result  of  a  training  program  and  answers  the 
question  of  improved  performance  in  another  environment,  on- 
the-job-  setting.  As  noted  by  other  researchers. 

The  effects  of  training  on  unit  performance  and 
operational  readiness  provide  the  final  measure  of 
training  effectiveness  and  represent  the  end  products  of 
the  maintenance  training  system  (Gibson  and  Orlansky, 
1986,  pp.48) . 
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One  advantage  to  using  information  from  the  NALDA  system 
is  that  it  provides  an  objective  measure  of  real  maintenance 
performance.  Since  data  bank  information  is  collected 
routinely  and  unobtrusively,  it  represents  actual  performance 
as  opposed  to  data  collected  under  test  conditions  that  may  be 
subject  to  the  "Hawthorne  Effect"  (Gibson  and  Orlansky,  1986, 

pp.26)  . 

However,  the  disadvantage  is  that  the  service's 
maintenance  data  was  not  specifically  designed  for  training 
purposes.  To  be  useful  for  evaluating  the  effect  of  training 
on  an  individual,  it  is  necessary  to  be  able  to  clearly  relate 
the  maintenance  data  to  the  specific  system,  work  center,  and 
the  performing  technician.  The  NALDA  information  does  not 
identify  individual (s)  who  was (were)  actually  involved  in  the 
performance  of  a  maintenance  task.  Therefore,  the  objective 
measures  of  performance  collected  by  this  method  only  reflect 
the  group  as  a  whole. 

Correlation  analyses  are  conducted  between  the  results  of 
the  MTIP  completion  rates  and  the  selected  performance 
measures  from  the  NALDA  data.  It  is  reasonable  to  believe 
that  well -trained  personnel  perform  better  on  their  jobs  than 
less -well -trained  personnel.  It  is  assvimed  here  that  higher 
MTIP  completion  rates  are  related  to  better- trained  personnel. 
If  this  is  so,  the  effects  of  better  training  should  be 
observable  in  the  selected  NALDA  operational  measures. 
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1 .  Participants 

During  the  planning  phase  of  this  study,  the  author 
examined  the  types  of  aircraft  that  are  deployable  aboard  an 
aircraft  carrier.  The  west  coast  carrier  deployable  aircraft 
are  home-based  at  several  Naval  Air  Stations  (NASs)  in  the 
State  of  California,  including  NAS  Lemoore,  NAS  North  Island 
and  NAS  Miramar,  and  at  NAS  Whidbey  Island  located  in  the 
State  of  Washington.  Preliminary  information  was  requested 
from  the  Maintenance  Training  Improvement  Program  Offices  and 
the  Data  Analysts  of  the  Pacific  Fleet  Light  Attack  Wing 
(LATWINGPAC)  at  NAS  Lemoore,  Anti-Submarine  Warfare  Wing 
(ASWWINGPAC)  at  NAS  North  Island,  and  Fighter  Airborne  Early 
Warning  Wing  (PITAEWWINGPAC)  at  N7^  Miramar.  The  author 
understands  the  extra  workload  that  such  requests  place  on 
personnel  at  these  units.  Thus,  it  is  not  surprising  that 
cooperation  was  sporadic.  However,  the  MTIP  Office  and  the 
Data  Analyst  from  the  PITAEWWINGPAC  have  been  extremely 
cooperative  and  helpful.  Therefore,  F-14  (Tomcat)  and  E-2 
(Hawkeye)  squadrons  under  the  Fighter  Airborne  Early  Wing, 
Pacific  Fleet  at  Naval  Station  Miramar  were  selected  as  the 
subjects  of  this  thesis. 

Once  the  types  of  aircraft  were  selected,  the  emphasis  was 
placed  on  the  sea-going  squadrons  to  be  consistent  with  the 
types  of  operational  cycles  involved  (e.g.,  post - deployment , 
turnaround  training,  and  deployment) .  Ten  F-14  squadrons  and 
six  E-2  squadrons  are  considered  to  be  sea-going  units. 
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However,  two  F-14  squadrons  and  one  E-2  squadron  are  currently 
assigned  to  USS  Independence,  homeported  at  Naval  Station 
Yokosuka,  Japan.  These  three  squadrons  are  excluded  from  this 
study  because  of  their  unique  operational  tempo,  which  is 
quite  different  from  the  rest  of  the  squadrons.  Therefore, 
the  total  number  of  squadrons  involved  in  this  study  is 
thirteen. 

An  average  of  180  and  110  enlisted  maintenance  technicians 
are  respectively  assigned  to  each  F-14  and  E-2  squadron. 
After  subtracting  fifteen  percent  (a  conservatively  high 
estimate)  of  E-7  and  adjove  enlisted  personnel  from  each 
squadron,  the  number  of  E-6  and  below  maintenance  technicians 
is  estimated  to  be  around  seventeen  hundred. 

Since  all  maintenance  personnel  (E-6  and  below)  are 
required  to  be  tested  under  the  Maintenance  Training 
Improvement  Program,  the  age  bracket  of  these  personnel  ranges 
from  19  years  old  to  the  early  30s.  There  are  no  females 
currently  assigned  to  sea-going  squadrons  deployable  aboard 
aircraft  carriers  in  the  Navy.  With  recent  changes  in  the 
political  climate,  this  will  certainly  change  in  the  near 
future . 

Due  to  the  method  of  data  collection  from  the  two 
centralized  sources,  none  of  these  maintenance  personnel, 
squadrons,  nor  the  FITAEWWINGPAC  staffs  are  aware  of  this 
experiment.  This  is  done  to  hopefully  control  some  of  the 
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internal  and  external  variables  which  may  influence  the 
evaluation  process, 

2 .  Materials  Used 

a.  Source  of  Data 

The  data  for  this  study  came  from  two  of  the  Navy's 
data  banks:  Aviation  Training  Support  System  (ATSS)  and  Naval 
Aviation  Logistics  Data  Analysis  (NALDA)  System.  Maintenance 
Training  Improvement  Program  records  are  managed  by  the  ATSS 
located  at  Naval  Air  Station  Miramar.  Operational  readiness 
data  are  obtained  from  the  NALDA  System,  Patuxent  River, 
Maryland  via  a  computer  terminal  with  the  assistance  of  the 
FITAEWWINGPAC  Data  Analyst  at  NAS  Miramar. 

Due  to  data  processing  and  storage  limitations  of  ATSS, 
most  squadron  MTIP  records  are  only  kept  to  cover  a  period  of 
a  couple  of  years.  Therefore,  the  historical  files  including 
the  messages  issued  monthly  by  the  FITAEWWINGPAC  MTIP  Office 
during  the  period  of  January  1989  to  December  1992  are  used  as 
the  source  of  the  MTIP  completion  rates.  As  a  result  of  this 
limitation,  operational  data  from  NALDA  has  also  been  selected 
to  match  the  available  MTIP  records  covering  the  same  period. 

The  total  number  of  pages  of  NALDA  information  obtained 
for  this  study  is  estimated  to  be  about  1200  pages  of  full 
size  computer  printouts.  Even  with  the  assistance  of  a 
senior,  experienced  data  analyst,  the  NALDA  data  took  more 
than  two  months  to  collect.  First  of  all,  this  is  partly  due 
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to  the  tremendous  amount  of  time  required  to  get  connected 
with  the  NALDA  system.  The  data  has  to  be  requested  by  each 
squadron,  and  by  each  month  for  which  it  is  needed,  in  the 
format  desired  and  it  has  to  be  printed  via  telephone  line 
with  computer  hook-up  to  the  source.  Secondly,  the  data 
analyst  had  to  do  all  of  this  on  a  voluntary  basis  and  very 
often  on  his  own  time. 

b.  Heurducure  and  Software 

A  standard  IBM  PS/2  286-based  micro -computer,  model  60 
with  one  megabyte  of  RAM  and  Disk  Operating  System  (DOS) 
Version  5.0  served  as  the  work  station  for  this  study.  This 
system  is  probaibly  considered  antiquated  compared  to  what  is 
available  on  the  market  today.  However,  for  conducting  a 
routine  statistical  analysis,  this  system  serves  the  purpose 
more  than  adequately. 

The  Lotus  1-2-3  (DOS  Version  2.2)  spreadsheet  software 
program  was  used  to  discriminate,  sort,  and  compile  the  1200 
pages  of  raw  data  to  a  form  relevant  for  the  periods  under 
consideration.  These  files  were  used  as  inputs  to  a  second 
software  program,  Statgraphics  (DOS  Version  6.0) ,  for  the 
correlation  analysis.  Lastly,  WordPerfect  (DOS  Version  5.0) 
was  used  to  document  the  entire  process. 
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C.  PROCEDURE 


Now  that  the  methodology,  participants,  and 
software/hardware  have  been  discussed  in  detail,  the  steps  in 
the  execution  of  this  analysis  can  be  summarized  as  follows: 

1.  Collect  ^f^IP  completion  rates  and  NALDA  operational 
measures  for  each  squadron. 

2.  To  ensure  comparability,  the  MTIP  data  has  to  be 
compiled  in  parallel  with  each  squadron's  turnaround  training 
cycle  (from  post -deployment  test  to  pre- deployment  test)  and 
the  NALDA  data  has  to  cover  the  time  period  following  the  pre- 
deployment  test  to  the  next  post -deployment  test.  The 
assumption  is  that  during  deployment  (including  a  month  or  two 
before  and  after  training)  each  squadron  is  provided  with 
maximum  support  from  up  and  down  the  chain  of  command.  If 
absolutely  necessary,  the  support  will  come  in  the  form  of 
temporarily  sacrificing  or  limiting  the  operational 
capabilities  of  other  similar  commands  not  scheduled  for 
deployment.  Therefore,  squadrons  on  deployment  will  most 
likely  be  as  closely  comparable  as  possible,  '  In  this  manner 
the  effects  of  training  are  measured  with  operational 
readiness  under  similar  conditions. 

3.  MTIP  completion  rates  represent  training  efficiency  of 
each  squadron  as  a  whole  with  one  number  representing  the 
total  effort  of  the  entire  turnaround  training.  However, 
operational  readiness  figures,  which  are  also  group  measures 
for  each  squadron,  are  collected  on  a  monthly  basis  throughout 
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the  deployment.  Therefore,  the  operational  readiness  data 
needs  to  be  averaged  out  so  that  it  may  be  compared  with  the 
MTIP  data.  To  ease  the  management  of  these  data,  Lotus  1-2-3 
was  used  to  arrange/ rearrange,  compile,  and  calculate  the 
information  required. 

4.  Import  the  compiled  data  from  Lotus  1-2-3  to  the 
Statgraphics  program  and  conduct  correlation  analyses. 

D.  LIMITATIONS 

1.  Available  Data 

As  mentioned  earlier  in  this  chapter,  non-availability 
of  the  MTIP  data  over  an  extended  period  of  time  limits  this 
study  to  a  period  of  four  years.  Considering  that  most  of  the 
MTIP  offices  and  the  Aviation  Training  Support  System  (ATSS) 
retain  information  for  only  a  couple  of  years,  it  is  fortunate 
to  be  able  to  obtain  the  data  covering  the  period  from  Jan 
1989  to  Dec  1992.  This  cunount  of  data  would  seem  sufficient 
to  conduct  an  assessment  of  the  Maintenance  Training 
Improvement  Program  (MTIP) . 

However,  squadrons  deploy  at  different  times  during  that 
four  years.  Some  squadrons  have  a  much  longer  turnaround 
training  cycle  than  others,  due  to  the  transition  to  a  newer 
model/type  of  aircraft  or  major  configuration  changes  to  the 
existing  ones.  A  few  squadrons  are  permanently  stationed  in 
Japan  with  different  operating  cycles  than  those  in  the 
continental  United  States.  Some  were  in  the  middle  of  a 
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deployment  when  the  December  1992  data  were  collected.  For 
these  various  reasons,  on  an  average,  each  squadron  has  only 
one  complete  set  of  MTIP  and  NALDA  data  usable  for  this  study. 

2.  Internal  and  External  Validity 

Due  to  the  four-year  period  of  time  involved,  specific 
events  other  than  training  occurring  between  sequential 
measurements  were  not  subject  to  control.  Biological  or 
psychological  effects  that  systematically  vary  with  the 
passage  of  time  as  participants  become  older,  fatigued,  or 
more  or  less  interested  in  the  job  between  measurements  were 
also  not  subject  to  control. 

Influence  of  the  pretest  on  the  scores  of  the  posttest 
and  the  reactive  effects  of  the  pre- deployment  test  which  can 
lead  to  increased  sensitivity  to  the  instructional  procedure 
were  negligible  because  of  the  many  months  in  between  each 
test  (6  to  8  months  for  post -deployment  test  and  12  to  18 
months  for  pre -deployment  test) . 

Rivalry  between  squadrons  does  exist.  Sister 
squadrons  that  have  the  same  deployment  schedule  under  the 
same  Air  Wing  often  put  up  extra  efforts  to  outperform  the 
other.  The  MTIP  office  makes  unannounced  visits  to  those 
squadrons  that  are  either  doing  too  well  or  not  too  well  to 
substantiate  the  reasons  behind  the  success  or  failure.  Self- 
serving,  innovative  ways  of  improving  the  MTIP  completion  rate 
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is  thus  kept  to  a  minimum.  It  is  impossible  to  consistently 
manipulate  completion  rates  or  operational  readiness  measures. 

Influences  of  the  interactions  between  variables  upon 
an  evaluation  are  more  difficult  to  understand  and  control. 
It  is  not  clear  as  to  how,  and  in  what  ways,  such  interactions 
affected  this  evaluation. 

Other  variables  (see  Chapter  II  for  an  explanation  of 
these  threats)  such  as  instrumentation,  differential  selection 
of  participants,  and  reactive  effects  of  experimental  settings 
are  well  under  control.  The  Maintenance  Training  Improvement 
Program  (MTIP)  is  mandatory  for  all  E-6s  and  below.  Each 
squadron  operates  under  the  same  policy  and  guidance  set  forth 
by  the  highest  level  of  the  chain  of  command  with  the  MTIP 
office  serving  as  the  monitoring  activity.  Squadrons  going 
through  major  configuration  or  model  changes  are  retested  by 
the  MTIP  office  based  on  the  new  information  and  are 
selectively  excluded  from  this  study. 

E.  RESULTS 

Based  on  the  consolidated  data  collected  from  the  eight 
F-14  and  five  E-2  squadrons,  this  section  provides  the 
correlation  analyses  between  the  MTIP  completion  rates  and  the 
selected  operational  measures.  The  procedure  generates  a 
matrix  of  correlation  coefficients  for  the  observed  values, 
and  provides  a  preliminary  view  of  the  relationships  among 
variables.  The  strength  of  the  relationship  between  variables 
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are  evaluated  using  the  Pearson  Product  Moment  Correlation 
Coefficient.  The  coefficient  values  fall  between  -1  and  +1. 
A  positive  correlation  indicates  that  the  variables  vary  in 
the  same  direction  while  a  negative  correlation  indicates  that 
the  variables  vary  in  the  opposite  direction.  Statistically 
independent  variables  have  an  expected  correlation  of  zero. 

Tables  1  and  2  provide  a  synopses  of  the  F-14  and  E-2 
aircraft  community  consolidated  data,  respectively,  collected 
by  each  squadron  on  the  MTIP  completion  rate.  Full  Mission 
Capable  (FMC)  rate,  number  of  no- defect  maintenance  (A- 799) 
actions,  and  Direct  Maintenance  Manhours  per  Flight  Hour 
(DMMH/FH)  .  In  order  not  to  disclose  the  name  of  each  aviation 
unit,  numbers  are  used  to  represent  squadrons. 


TABLE  1  P-14  CONSOLIDATED  DATA 


Squadron 

No. 


MTIP 

Conqpletlon 

Rate 

(Percent) 

Pull  Mission 
Capable 

Rate 

(Percent) 

Number 

of 

A-799 

Actions 

Direct 
Malnt . 
Manhours 
per  Pit  Hr 
(DMMH/PH) 

57 

56 

69 

36 

75 

62 

61 

45 

73 

61 

64 

58 

75 

63 

63 

38 

31 

33 

58 

52 

65 

36 

50 

56 

50 

43 

52 

29 

62 

35 

43 

92 

Note:  Detailed  F-14  squadrons  month -by -month  operational 
figures  can  be  found  in  Appendix  C. 


Source:  Compiled  from  the  MTIP  Office  historical  files  and  HALDA  data. 
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TABLE  2  E-2  CONSOLIDATED  DATA 


Squadron 

No. 

MTIP 

Coiiq>letlon 

Rate 

(Percent) 

Full  Mission 
Capable 

Rate 

(Percent) 

Nunber 

of 

A-799 

Actions 

Direct 
Malnt . 
Manhours 
per  Fit  Hr 
(DMMH/FH) 

1 

37 

30 

46 

26 

2 

66 

60 

34 

24 

3 

41 

67 

42 

22 

4 

72 

57 

16 

23 

5 

90 

68 

21 

24 

Note:  Detailed  E-2  squadrons  month -by -month  operational 

figures  can  be  found  in  Appendix  D. 

Source:  Compiled  from  the  MTIP  Office  historical  files  and  NALDA  data. 


Table  3  provides  a  general  overview  cE  the 
intercorrelation  statistics  obtained,  based  on  the  procedure 
discussed  in  this  chapter. 


TABLE  3  INTERCORRELATION  MATRIX 


MTIP 

FMC 

A-799 

DMMH/FH 

MTIP 

♦ 

0.61 

-0.26 

-0.02 

FMC 

0.61 

* 

-0.15 

-0.45 

A-799 

-0.26 

-0.15 

★ 

0.36 

DMMH/FH 

-0.02 

-0.45 

0.36 

★ 

Notes : 

MTIP  Maintenance  Training  Improvement  Program 

FMC  Full  Mission  Capable 

A-799  Number  of  no-defect  maintenance  actions 

DMMH/FH  Direct  Maintenance  Manhours /Flight  Hour 

1.  Relationship  of  MTIP  to  PMC 

Correlation  between  the  Maintenance  Training 
Improvement  Program  (MTIP)  con^letion  rate  and  the  Full 
Mission  Capable  (FMC)  rate  shows  that  there  is  a  moderate 
positive  relationship  between  the  two  variadjles.  As  reflected 
in  Table  3,  the  correlation  coefficient  is  0.61.  The 
probability  of  a  relationship  this  strong  occurring  by  chance 
is  0.03  as  determined  by  a  t-test.  In  other  words,  the  Full 
Mission  Capable  rate  is  positively  associated  with  the  MTIP 
completion  rate.  As  the  MTIP  completion  rate  increases,  the 


FMC  rate 

also 

tends 

to 

increase  in 

the  same 

direction. 

However, 

this 

does 

not 

mean  that 

there  is 

a  causal 

relationship  between  the  two  variables.  We  can  only  infer 
that  the  two  variables  have  a  moderate  tendency  to  increase  or 
decrease  simultaneously. 

2.  Relationship  of  MTIP  to  A- 7 99 

Correlation  between  the  Maintenance  Training 
Improvement  Program  (MTIP)  and  the  number  of  no-defect  (A- 799) 
maintenance  actions  produces  a  weak  negative  relationship. 
The  MTIP  completion  rate  and  the  number  of  A- 799  maintenance 
actions  have  a  correlation  coefficient  of  -0.26  (see  Table  3) . 
The  correlation  is  statistically  insignificant.  The 
probability  of  a  relationship  occurring  by  chance  between 
these  two  variables  is  about  0.39.  A  negative  correlation 
coefficient  suggests  that  the  variables  tend  to  increase  or 
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decrease  in  the  opposite  directions  simultaneously.  However, 
a  negative  correlation  between  the  MTIP  completion  rate  and 
the  number  of  no-defect  maintenance  actions  is  expected.  Non- 
faulty  parts  are  those  that  were  removed  but  found  not  to  be 
defective  when  tested  later.  Previous  research  does  offer 
some  evidence  that  maintenance  technicians  may  conduct 
maintenance  in  a  more  appropriate  and  efficient  manner  as 
maintenance  technicians  learn  through  a  training  program  and 
become  more  knowledgeable.  Thus,  the  results  found  here  are 
in  the  expected  direction  but  not  strong  enough  to  be 
conclusive. 

3.  Relationship  of  MTIP  to  DMMH/FH 

Even  though  it  is  reasonable  to  believe  that  with  more 
training  the  time  required  to  complete  a  maintenance  action 
will  be  reduced,  in  this  comparison,  there  does  not  appear  to 
be  any  relationship  between  the  MTIP  completion  rate  and  the 
Direct  Maintenance  Manhours  per  Flight  Hour  (DMMH/FH) .  The 
MTIP  completion  rate  and  the  DMMH/FH  have  a  negative 
correlation  coefficient  of  -0.02.  The  probability  of  a 
relationship  occurring  by  chance  between  these  two  variables 
is  about  0.94.  In  other  words,  MTIP  and  Direct  Maintenance 
Manhours  per  Flight  Hour  are  statistically  independent  with  a 
correl  ion  coefficient  of  nearly  zero. 

As  reflected  in  Table  3,  the  correlation  between  the 
Direct  Maintenance  Manhours  per  Flight  Hour  (DMMH/FH)  and  the 
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niaxnber  of  A- 799  maintenance  actions  provides  positive  but  not 
statistically  significant  evidence  that  as  the  number  of 
inappropriate  or  inefficient  maintenance  practice  increases, 
the  Direct  Maintenance  Manhours  per  Flight  Hour  also  increases 
in  the  same  direction.  The  correlation  coefficient  is  0.36 
and  the  probability  of  a  relationship  this  strong  occurring  by 
chance  between  these  two  variadDles  is  adDout  0.23. 

The  correlation  between  the  DMMH/FH  and  the  Full 
Mission  Capable  (FMC)  rate  produces  a  moderately  negative 
relationship.  As  expected,  when  the  DMMH/FH  decreases,  the 
FMC  rate  improves.  The  correlation  coefficient  between  the 
DMMH/FH  and  the  FMC  rate  is  -0.45.  And,  the  probability  of  a 
relationship  this  strong  occurring  by  chance  between  these  two 
variables  is  at  aibout  0.12. 

Therefore,  as  seen  in  this  section,  the  Direct 
Maintenance  Manhours  per  Flight  Hour  could  possibly  be 
indirectly  related  to  the  MTIP  completion  rate  through  a  third 
variable  such  as  the  number  of  A- 799  no- defect  maintenance 
actions  or  the  FMC  rate.  The  relationships  between  the  MTIP 
completion  rate,  the  FMC  rate,  the  number  of  no-defect 
maintenance  actions  and  the  DMMH/FH  are  summarized  graphically 
in  Figure  1. 
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Figure  1.  Intercorrelation  Diagram 
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V.  CGNCLUSIOHS  AND  RECOMMENDATIONS 

A.  CONCLUSIONS 

Training  is  defined  as  the  systematic  acquisition  of 
skills,  rules,  concepts,  or  attitudes  that  result  in 
improved  performance  in  another  environment  (Goldstein, 
1993)  . 

The  purpose  of  the  Maintenance  Training  Improvement 
Progrcim  is,  therefore,  to  prepare  sailors  from  the  training 
environment  to  perform  the  technical  tasks  necessary  in  the 
real-world  environment  to  assure  the  availability  and  proper 
functioning  of  aircraft  weapons  systems.  The  ultimate  goal  is 
to  improve  unit  performance  in  terms  of  operational  readiness, 
without  systematic  evaluation,  there  is  no  feedback  to  provide 
information  necessary  to  enhance  and  improve  this  training 
program. 

Within  the  context  of  the  limitations  discussed  earlier, 
the  results  of  this  thesis  have  provided  empirical  support  for 
the  idea  that  aviation  maintenance  training  under  the  MTIP 
management  system  may  have  an  impact  on  unit  performance.  The 
MTIP  completion  rate  does  show  a  tendency  to  increase  or 
decrease  simultaneously  with  one  of  the  most  important 
operational  readiness  measures- -the  FMC  rate.  The  MTIP 
completion  rate  is  a  moderate  predictor  of  the  Full  Mission 
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Capable  rate;  as  more  training  is  accomplished,  the  FMC  rate 
tends  to  improve  in  the  positive  direction. 

The  MTIP  has  also  been  shown  to  have  a  negative 
relationship  with  the  total  number  of  no-defect  maintenance 
actions  performed  by  the  maintenance  technicians.  Well- 
t rained  maintenance  technicians  are  less  likely  to  conduct 
maintenance  in  an  inappropriate  and  inefficient  manner. 
Therefore,  fewer  non- faulty  parts  will  be  removed  and  found  to 
be  operational  later. 

The  MTIP  completion  rate  appears  to  have  no  correlation 
with  the  Direct  Maintenance  Manhours  per  Flight  Hour 
(DMMH/FH) .  However,  as  reflected  in  Table  3  and  Figure  1  of 
Chapter  IV,  the  MTIP  completion  rate  is  moderately  and 
positively  correlated  to  the  FMC  rate,  and  only  weakly  and 
negatively  correlated  to  the  A- 799  actions.  But  at  the  same 
time,  the  DMMH/FH  reveals  that  it  is  also  moderately 
correlated  to  the  A- 799  actions  and  the  FMC  rates.  Therefore, 
it  is  possible  that  the  MTIP  completion  rate  is  somehow 
indirectly  related  to  the  DMMH/FH  through  a 'third  variable 
such  as  the  number  of  A- 799  maintenance  actions  or  the  FMC 
rate.  This  is  an  area  that  could  be  explored  further. 

In  interpreting  the  results  of  the  research  conducted  for 
this  thesis,  using  correlation  analyses,  some  insight  has  been 
gained  in  explaining  the  question  raised  in  the  introductory 
chapter  on  whether  the  effects  of  training  are  observable  in 
operational  indicators. 
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B.  RBCOMMBNDATIONS 

This  thesis  has  just  touched  the  surface  of  evaluating  the 
Maintenance  Training  Improvement  Program  and  has  disclosed 
several  areas  where  there  is  room  for  more  in-depth  follow-up 
research.  Hopefully,  this  paper  will  serve  as  the  catalyst  to 
ultimately  enhance  and  improve  the  aviation  training  progrcun 
under  the  MTIP  management  system. 

Until  now,  the  effectiveness  of  MTIP  training  has  been 
evaluated  primarily  by  end -of -course  tests  with  the  MTIP 
completion  rate  as  the  final  group  measure.  Unless  we  have 
credible  information  aibout  how  well  technicians  perform  after 
training  in  the  real -world- setting,  we  do  not  know  very  much 
about  whether  the  course  provided  the  information  needed  to 
perform  well  on  the  job.  This  thesis  has  introduced  some  of 
the  available  operational  measures  to  link  training  with 
operations.  However,  to  better  evaluate  MTIP,  operational 
measures  should  be  directly  traceable  to  an  individual  instead 
of  a  group.  Modifications  to  the  existing  NALDA  data  base  may 
be  required.  Use  of  local  historical  files  is"  another  option 
to  trace  an  individual's  training  and  his/her  on-the-job 
performance.  Measures  other  than  the  MTIP  completion  rate, 
such  as  job- sample  tests  and  on- equipment  performance  tests, 
also  need  to  be  explored. 

As  observed  in  this  study,  collection  and  processing  of 
the  required  data  can  be  extremely  time  consuming.  With  some 
training  and  proper  software  packages,  the  Data  Analysts  and 
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I 

I 

the  Wing  Training  Officers  could  systematically  collect  the 
necessary  data  on  a  continual  basis.  They  could  also  perform 
simple  analyses  on  their  own  to  draw  inferences,  revealing  the 
possible  weaknesses  and  strengths  of  a  training  program,  or 
make  these  data  available  for  further  study  by  others  like 
students  at  the  Naval  Postgraduate  School . 

! 

It  is  recommended  that  the  Aviation  Training  Support 
System  (ATSS)  database  storage  capacity  and  the  local  MTIP 
Office  files  be  expanded  to  retain  historical  files  over  a 
longer  period  of  time.  It  will  be  extremely  useful  to  be  able 
to  track  training  performance  over  several  complete  cycles  of 
training  and  deployment. 

It  is  not  clear  at  this  point  why  the  MTIP  completion  rate 
and  the  Direct  Maintenance  Manhours  per  Flight  Hour  did  not 
produce  the  expected  correlation.  On  one  hand,  the  result  may 
be  a  factor  of  the  particular  data  captured  here  and  not 
generalizable  to  other  periods  of  time  or  other  squadrons.  On 
the  other  hand,  if  the  data  reflect  weakness  in  the  training, 
this  is  an  important  issue.  Another  possibility  is  that  the 
zero  correlation  is  a  consequence  of  mixing  data  for  two 
different  aircraft  types.  Further  research  should  be 
conducted  to  explore  this  finding. 

Another  research  area  that  should  be  explored  is  the 
correspondence  between  the  objectives  of  MTIP  and  actual  job 
performance.  Is  there  100  percent  correspondence  between  what 


57 


is  taught  and  what  is  expected  on  the  job?  Until  this 
analysis  is  conducted,  further  evaluation  may  be  meaningless. 

In  view  of  the  results  of  this  thesis  in  empirically 
linking  the  west  coast  F-14  and  E-2  aircraft  community 
Maintenance  Training  Improvement  Prograun  (MTIP)  completion 
rates  with  the  operational  readiness  measures  s’^ch  as  the  Full 
Mission  Capable  (FMC)  rates  and  the  number  of  no-defect  (A- 
799)  maintenance  actions,  the  next  logical  step  would  be  to 
conduct  a  study  involving  all  types  of  aircraft  in  the  Navy. 
Other  exploratory  studies  will  add  information  that  can  be 
used  to  ultimately  determine  the  training  effectiveness  of 
MTIP.  Resources  allocated  to  MTIP  and  personnel  safety  make 
adequate  evaluation  critical. 
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APPENDIX  A 


AVAILABLE  MTIP  REPORTS 


1  Report  Nfumber 

Title  1 

MOl 

MTIP  Personal  Training  Report 

MO  2 

MTIP  Work  Center/  System  Training 

MO  3 

MTIP  Class  Status 

MO  4 

MTIP  Course  Summary  Report 

MOB 

MTIP  Squadron/Work  Center/CDP  Report 

MO  6 

MTIP  Wing  Test  Area  Report  by  Work 
Center/System 

MO  7 

MTIP  Squadron  Test  Area  Report  by  Work 
Center/System 

MO  8 

MTIP  Wing  Work  Center/System  Report  by 
Squadron 

MO  9 

MTIP  Wing  Test  Area  Report  by  Squadron 

MIO 

MTIP  Squadron  Training  Requirements 

Reports 

Mil 

MTIP  Top  Wing  System  Training  I 

Requirements  | 

Ml  3 

MTIP  Gradebook  Random  Report 

Ml  5 

MTIP  Test  Item  Data  Random  Review 

Ml  7 

MTIP  Work  Center/System  Report 

Ml  8 

MTIP  Post/Pre- Cruise  Training  Progress 
Report 

Ml  9 

Functional  Wing  Refresher  Training 

Progress 

M21 

Squadron  Turnaround  Training  Progress 

Report 

APPENDIX  B 


MTIP  COMPLETION  RATES 


MTIP  RATES  (%) 


57 

75 

73 

75 

31 

65 

50 

62 


F-14  Squadrons 


1 

2 

3 

4 

5 

6 

7 

8 


MTIP  RATES  (%) 


37 

66 

41 

72 

90 


E-2  Squadrons 


1 

2 

3 

4 

5 


60 


APPENDIX  C 


F'14  OPERATIONAL  DATA 


Squadron  No . 1 


DATE 

%FMC 

A-799 

DMMH/FH 

II 

II 

II 

II 

II 

II 

II 

II 

II 

II 

H 

N 

II 

II 

II 

II 

II 

=  tt  =  sa;s  =  =r  = 

=  K  =  =  s  =  s  =  a  =  i 

1990&91 

OCT 

38 

59 

28 

NOV 

N\ 

76 

NA 

DEC 

45 

86 

42 

JAN 

64 

115 

21 

FEB 

61 

80 

18 

MAR 

66 

77 

24 

APR 

67 

73 

36 

MAY 

60 

43 

48 

JUN 

47 

9 

73 

Avg 

56 

69 

36 

Squadron  No. 2 


DATE 

%FMC 

A-799 

DMMH/FH 

1991 

JAN 

50 

85 

22 

FEB 

62 

74 

20 

MAR 

63 

79 

41 

APR 

80 

66 

42 

MAY 

63 

43 

50 

JUN 

53 

17 

93 

Avg 

62 

61 

45 

61 


Scjuadron  No .  3 


DATE  %FMC  A- 799 


1990 


FEB 

60 

23 

MAR 

NA 

42 

APR 

47 

64 

MAY 

30 

59 

JUN 

52 

75 

JUL 

73 

64 

AUG 

77 

98 

SEP 

62 

102 

OCT 

59 

96 

NOV 

69 

49 

DEC 

84 

30 

Avg 

61 

64 

Squadron  No. 4 


DATE 

%FMC 

A- 799 

1990&91 

DEC 

53 

37 

JAN 

52 

62 

FEB 

56 

56 

MAR 

67 

107 

APR 

66 

81 

MAY 

59 

33 

JUN 

65 

NA 

JUL 

76 

NA 

AUG 

86 

NA 

SEP 

49 

NA 

Avg 

63 

63 

DMMH/FH 


73 

NA 

31 

51 

57 

48 

63 

51 

59 

86 

59 


58 


DMMH/FH 


26 

36 

41 

27 

35' 

25 

26 
36 
82 
44 


38 


62 


♦Squadron  No. 5* 


Note; 


DATE 

%FMC 

A-799 

DMMH/FH 

1992 

MAY 

57 

NA 

49 

JUN 

62 

NA 

30 

JUL 

57 

NA 

35 

AUG 

54 

NA 

49 

SEP 

42 

NA 

99 

OCT 

42 

NA 

53 

NOV 

65 

NA 

79 

DEC 

57 

NA 

41 

Avg 

55 

NA 

54 

Squadron  No 

.5  is 

assigned  to 

USS  Independence 

Squadron  No. 6 

DATE  %FMC  A- 799  DMMH/FH 


1989&1990 


NOV 

44 

52 

75 

DEC 

52 

19 

60 

JAN 

38 

62 

98 

FEB 

26 

91 

33 

MAR 

34 

73 

23 

APR 

24 

54 

33 

MAY 

10 

65 

44 

JUN 

NA 

46 

NA 

Avg 

33 

58 

52 

63 


r 


Squadron  No . 7 

DATE 

%FMC 

A-799 

DMMH/FH 

1991 

MAR 

15 

70 

48 

APR 

29 

51 

47 

MAY 

44 

49 

27 

JUN 

50 

46 

68 

JUL 

29 

85 

63 

AUG 

27 

72 

43 

SEP 

32 

59 

40 

OCT 

46 

43 

51 

NOV 

61 

18 

62 

DEC 

23 

8 

116 

Avg 

36 

50 

56 

♦Squadron  No. 8* 


DATE 

%FMC 

A-799 

DMMH/FH 

II 

u 

II 

II 

II 

II 

II 

II 

sssssas  =  s  =  =  s  = 

=  =  =  =  »  =  =  =  : 

1990 

FEB 

50 

57 

40 

MAR 

100 

56 

44 

APR 

33 

54 

44 

MAY 

61 

109 

36 

JUN 

31 

56 

49  ■ 

JUL 

29 

44 

47 

AUG 

34 

21 

46 

SEP 

50 

40 

44 

OCT 

68 

50 

35 

NOV 

75 

63 

55 

DEC 

88 

37 

40 

Avg 

56 

587 

43 

Note:  Squadron  No. 8  is  assigned  to  USS  Independence. 
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Squadron  No. 9 


DATE 

%FMC 

1990&91 

DEC 

23 

JAN 

26 

FEB 

26 

MAR 

54 

APR 

54 

MAY 

47 

JUN 

58 

JUL 

35 

AUG 

72 

SEP 

33 

Avg 

43 

A- 799  DMMH/FH 


28 

31 

68 

29 

35 

49 

61 

24 

66 

22 

61 

23 

89 

27 

57 

29 

25 

33 

30 

26 

52 

29 

Squadron  No. 10 


DATE 

%FMC 

A-799 

DMMH/FH 

1991 

MAR 

44 

64 

47 

APR 

25 

63 

46 

MAY 

36 

42 

35 

JUN 

44 

43 

58 

JUL 

43 

73 

70 

AUG 

18 

50 

60  ' 

SEP 

24 

39 

49 

OCT 

37 

29 

85 

NOV 

55 

19 

91 

DEC 

25 

8 

377 

Avg 

35 

43 

92 

65 


APPENDIX  D 


E-2  OPERATIONAL  DATA 


Squadron  No . 1 


DATE 

%FMC 

A- 799 

DMMH/FH 

sESSESEsssa: 

II 

II 

II 

II 

II 

M 

H 

II 

II 

II 

II 

II 

II 

u 

II 

II 

II 

II 

II 

1991 

JAN 

40 

51 

19 

FEB 

29 

26 

29 

MAR 

37 

69 

lb 

APR 

20 

55 

24 

MAY 

31 

68 

30 

JUN 

NA 

83 

NA 

JUL 

NA 

33 

NA 

AUG 

35 

19 

21 

SEP 

18 

6 

38 

Avg 

30 

46 

26 

Squadron  No. 2 


DATE 

%FMC 

A-799 

DMMH/FH 

1990 

FEB 

42 

22 

38 

MAR 

100 

NA 

26 

APR 

46 

44 

25 

MAY 

53 

24 

18 

JUN 

53 

21 

26 

JUL 

NA 

48 

NA 

AUG 

44 

44 

24 

SEP 

NA 

62 

NA 

OCT 

NA 

34 

NA 

NOV 

66 

24 

19 

DEC 

73 

17 

16 

Avg 

60 

34 

24 

66 


Squadron  No . 3 


Note; 


DATE 

%FMC 

A- 799 

DMMH/FH 

=!  =  a  =  s==:;=s=s  = 

II 

II 

II 

II 

II 

II 

II 

II 

II 

1989&90 

NOV 

82 

19 

24 

DEC 

62 

29 

29 

JAN 

75 

36 

25 

FEB 

72 

49 

24 

MAR 

71 

67 

15 

APR 

48 

39 

15 

MAY 

53 

55 

20 

JUN 

75 

42 

25 

Avg 

67 

42 

22 

♦Squadron  No. 4* 


DATE 

%FMC 

A- 799 

DMMH/FH 

1990&91 

DEC 

75 

42 

35 

JAN 

22 

19 

71 

FEB 

32 

57 

33 

MAR 

44 

32 

18 

APR 

38 

31 

23  ” 

MAY 

57 

39 

59 

JUN 

55 

43 

21 

JUL 

75 

38 

24 

AUG 

77 

37 

19 

SEP 

73 

37 

25 

OCT 

99 

7 

108 

NOV 

43 

24 

63 

DEC 

72 

10 

23 

Avg 

59 

416 

40 

Squadron  No. 4  is  assigned  to  USS  Independence. 
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Squadron  No . 5 


DATE  %FMC  A- 799  DMMH/FH 


1990&91 


OCT 

51 

21 

19 

NOV 

NA 

26 

NA 

DEC 

47 

25 

24 

JAN 

50 

4 

14 

FEB 

46 

2 

13 

MAR 

51 

26 

19 

APR 

55 

15 

19 

MAY 

73 

23 

21 

JUN 

85 

1 

56 

Avg 

57 

16 

23 

Squadron  No . 6 


DATE 

%FMC 

A- 799 

DMMH/FH 

II 

II 

II 

II 

II 

II 

II 

II 

II 

II 

II 

1991 

APR 

57 

16 

13 

MAY 

65 

23 

24 

JUN 

75 

20 

25  ■ 

JUL 

66 

34 

29 

AUG 

68 

23 

22 

SEP 

67 

18 

22 

OCT 

NA 

24 

NA 

NOV 

80 

12 

35 

Avg 

68 

21 

24 

68 
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